1. Field of the Invention
The present invention relates to a processing technology for a still image and a moving image in order to expand a dynamic range of an image to be displayed or printed.
2. Description of the Related Art
While the dynamic range of human eyes reaches the 6th to 10th power of 10 (including adaption), the dynamic range in a normal digital photograph is only around the 4th power of 10. Therefore even if an image is captured with setting a correct exposure, blown out highlights and blocked up shadows often appear.
A method of solving this problem is generating a high dynamic range photograph by taking a plurality of photographs with different exposure values in advance, and composing them (see Japanese Patent Application Laid-Open No. 2008-236726 and No. 2006-345509). In the case of moving images, a certain degree of blown out highlights or blocked up shadows can be decreased by adjusting the gamma curve in a dark portion and a bright portion, and this function is called “knee adjustment”.
A prior art of generating a photograph in a high dynamic range will be described with reference to FIG. 9, FIG. 10A and FIG. 10B.
FIG. 9 is a functional block diagram for generating an image in a high dynamic range according to a prior art. In FIG. 9, 106 denotes a functional block for multiplying a correctly exposed image by a ratio for use A, 107 denotes a functional block for multiplying an overexposed image by a ratio for use B, and 108 denotes a functional block for multiplying an underexposed image by a ratio for use C. 109 denotes an addition block for adding three images, and 110 denotes a gradation conversion block for decreasing or increasing a number of gradation levels of an image to a number of gradation levels which is appropriate for an output image.
Three types of input images (correct exposure, overexposure, underexposure) are input from an input unit, which is not illustrated. A gradation value of each input image is a value generated by digitizing an output voltage of an image sensor which captured an image with each of the three types of exposures, and corresponds to the later mentioned reference numbers 111 to 113 in FIG. 10A respectively.
Then the functional blocks 106, 107 and 108 multiply the three types of images (correct exposure, overexposure, underexposure) by individual ratios (ratio: A, ratio: B, ratio: C) respectively. These images are added by the addition block 109 (linear composition). In this case, gradation values at the correct exposure and the overexposure are added in a dark side, and gradation values at correct exposure and underexposure are added in a bright side. As a result, a number of gradation levels to be output for a brightness to be input increases (high dynamic range).
Finally a gradation conversion block 110 adjusts a number of gradation levels of the image. For this, a processing to match the number of gradation levels with a required number of gradation levels is performed. The gradation conversion block 110 is normally constituted by a multiplier.
In this way, a high dynamic range image is obtained.
FIG. 10A and FIG. 10B are graphs for describing conventional high dynamic range (HDR) images. FIG. 10A shows the brightness to be input to an image sensor and the output of the image sensor, and FIG. 10B shows the brightness to be input to an image sensor and a gradation value of the HDR image.
In FIG. 10A, the abscissa is the brightness to be input to the image sensor, and the ordinate is an output gradation value of the image sensor, which is generated by converting the output of the image sensor into digital data by an analog/digital converter.
To simplify explanation, an 8-bit analog/digital converter is used in this example, where the output gradation value is shown as 0 to 255 data. A required number of levels of gradation (bit width) may certainly be used instead. In FIG. 10A, the dotted line 111 shows a relationship between the brightness to be input to an image sensor and an output gradation value of the image sensor when the image is captured with a correct exposure. The dotted line 112 shows a relationship between the brightness to be input to an image sensor and an output gradation value of the image sensor when the image is captured with overexposure. The dotted line 113 shows a relationship between the brightness to be input to an image sensor and an output gradation value of the image sensor when the image is captured with underexposure. In FIG. 10A, in the case of the correct exposure, the output gradation value is 0 if the brightness is lower than the brightness 111D which corresponds to the output gradation value 0, and the output gradation value is 255 if the brightness is higher than the brightness 111L which corresponds to the output gradation value 255, although these are not shown in order to avoid making the graph complicated. In other words, the gradation values other than the 0 to 255 range are clipped. In the case of overexposure and underexposure as well, the gradation values other than the 0 to 255 range are clipped.
In the case of normal photography without generating an HDR image, if an image is captured with a correct exposure, the output gradation value always becomes 0 if the brightness is lower than the brightness 111D, where the blocked up shadows are generated. The output gradation value always becomes 255 if the brightness is higher than the brightness 111L, where the blown out highlights are generated. The range of the brightness 111D to 111L corresponds to the dynamic range.
In order to solve these problems, when an HDR image is generated conventionally, an image is captured with underexposure or overexposure, and the obtained image is processed as described below, so as to expand the dynamic range (that is, the brightness at which the blocked up shadows are generated is decreased, and the brightness at which the blown up highlights are generated is increased).
The conventional method for generating an HDR image will be described with reference to FIG. 10B. In FIG. 10B, the abscissa is the brightness to be input to an image sensor, and the ordinate is the output gradation value of the image sensor and the gradation value of an HDR added image. The dotted lines 111, 112 and 113 show the brightness to be input to an image sensor and the output gradation value of the image sensor in the case of capturing an image with a correct exposure, in the case of capturing an image with overexposure, and in the case of capturing an image with underexposure respectively, as mentioned above.
The generation of an HDR image is processed in the blocks described in FIG. 9. To simplify explanation, it is assumed that the ratios A, B and C described in FIG. 9 are all 1. The output gradation values of the image sensor (dotted lines 111, 112, 113) are added in the addition block 109. The output of the addition block 109 is called an “HDR added image” here. The bold solid line 114 in FIG. 10B indicates the gradation values of the HDR added image. The HDR added image is converted into a required number of gradation levels by the gradation conversion block 110.
In the case of the solid line 114, as shown in FIG. 10B, the brightness at which the blocked up shadows are generated decreases down to the brightness indicated by 114D, and the brightness at which the blown out highlights are generated increases up to the brightness indicated by 114L. In other words, in the case of an HDR image, the dynamic range is expanded compared with the image captured with the correct exposure shown in FIG. 10A.
An HDR added image takes values from 0 to 765, as shown by the solid line 114, hence it is preferable to multiply the gradation value by ⅓ in the gradation conversion block 110 if this image is displayed on a display apparatus having 256 gradation levels.
In FIG. 10B, the form of the solid line 114 indicates the characteristics of the HDR image. The form of the solid line 114 can be changed depending on the set values of overexposure and underexposure and the ratios A, B and C described in FIG. 9, and can be designed to be an optimum characteristic form in a required dynamic range.